The invention relates generally to a method and system for browsing visual content using rasterized representations and related browsing data. In particular, the invention relates to a method for rendering and transforming visual content on a server system based on the display attributes of a client device, and transmitting the transformed visual content for display on a client device with respect to related browsing data.
Rendering is the process of converting visual content such as electronic documents (e.g., word-processing documents), spreadsheets, Web pages, electronic forms and electronic mail (email) into an image format useful for generating a display. In some instances, a rendered image requires additional processing to convert it into data that can be viewed on a particular display device or class of display devices. For example, rendered image data can be represented in a canonical pixel resolution and format, which are then transformed into a bitmap for display on a particular display device or class of display devices.
The term xe2x80x9crasterizingxe2x80x9d as used herein describes the process of generating a displayable bitmap image from input data. Rasterizing can include a rendering step, when the input data is in a non-image (or xe2x80x9csourcexe2x80x9d) form. A bitmap includes any ordered sequence of pixel data defining a displayable image, and can include, for example, data representing bi-tonal, grayscale, RGB, hue and saturation, and chrominance and luminance values. Thus, each visual content element of an image or other displayable file is accessed, interpreted and rasterized as a bitmap according to the display attributes of a display surface, which receives the rendered image. The contents of the display surface are then xe2x80x9cpaintedxe2x80x9d onto a bitmap display device.
Client-side rendering is a common method for rendering visual content. The client system in a client-server architecture controls a display device that displays the rendered visual content for viewing by the user. The client system generally includes a processor, memory, and bitmap display, and may be, for example, a personal computer (PC). The client accesses and then renders and displays the visual content. If client-side rendering requires the client to download and execute dynamic content, then the client is susceptible to viruses and other unwanted interference from outside sources.
Content layout and design intent are important considerations in the rendering of visual content. Visual content, such as a Web page, is generally designed to convey specific information and achieve a desired viewer reaction. This design is implicitly embedded in the xe2x80x9ccodingxe2x80x9d of the visual content and its constituent component(s). Interpreting this coding is part of the rendering function. Typically, content layout is based on PC displays. Currently, there is relative uniformity in client processing power, memory, display attributes and software environment due, in part, to the widespread use of MICROSOFT WINDOWS and similar operating systems. Consequently, the display of visual content is relatively uniform for most viewers.
As wireless networking becomes widely accepted and new intelligent client devices (e.g., Personal Digital Assistants (PDAs) and cellular telephones) are introduced to the public, the reliance on client-side rendering may begin to decrease. Processing power, memory, and software implementation will soon vary significantly among different client devices, overturning long-standing assumptions about the preferred layout of visual content; the diversity of display formats and capabilities will require new methods of providing visual content to a viewer using such client devices.
Content transcoding is one approach that addresses some of the problems of display rendering on new classes of client devices. Transcoding is an operation in which a visual content element is converted from one source format to another, before a rasterizing (or rendering) function is performed. In content transcoding, constituent components of the visual content are analyzed, and a new set of constituent components is generated by a gateway server. Content transcoding is generally based on client-side display rendering of the xe2x80x9cre-mappedxe2x80x9d content. Re-mapping includes various conversions of the constituent components of the visual content to accommodate the rendering capabilities of the client device.
The transcoding function can include filtering or extractive steps, where certain types of encoded content are converted, transformed or removed from the derived source representation. Transcoding can also perform a complete translation from one source encoding format to another. Transcoding can be lossless (all of the visually significant encoding and data are preserved) or lossy (some portions are not preserved).
For example, an HTML document can be rendered by an HTML rendering function in one rasterizing mode. This HTML source can also be transcoded to a WML (Wireless Markup Language) format and then rasterized by a WML rendering function in a second rasterizing mode. The two different representations can be associated as a multi-modal set, based on their relationship to the original HTML-encoded visual content element.
Transcoding can also be used to generate a different version of the source visual content element using the same encoding format as the original. For example, an HTML document can be transcoded into another HTML document, while changing, translating or removing certain encoded data. For example, references to unwanted or objectionable content can be removed, automatic language translation can be applied to text components, or layout directives can be removed or changed to other layout directives.
Many current Web pages are defined in HyperText Markup Language (HTML). Some wireless telephones are programmed to render documents based on the Wireless Access Protocol (WAP). One type of transcoding gateway reads the HTML for the Web page and attempts to remap this into equivalent WAP content. The wireless telephone then uses client-side rendering to render the WAP content for viewing on its bitmap display.
Transcoding can significantly alter the layout or contents of the re-mapped visual content as compared to the original, requiring sophisticated algorithms to preserve as much of the original intent as possible. An alternative to transcoding is to represent the same visual content in multiple forms, each form appropriate for client-side rendering on a given class of client devices. While not as technically challenging as transcoding, the additional time and cost to maintain multiple versions of the same content can be substantial.
The present invention relates to display of visual content on a client device using rasterized representations of visual content. In one aspect of the invention, visual content is rendered on a server system, transformed into bitmaps compatible with the display attributes of a client device, and transmitted for display on the client device. The the server and client coordinate to perform, in effect, as a remote multi-level browsing system for displaying Web pages, e-mail, e-mail attachments, electronic document and forms, database queries and results, is drawings, presentations, and images at the client device. The approach is xe2x80x9cremotexe2x80x9d because the server does the rendering and the client provides the interface; xe2x80x9cmulti-levelxe2x80x9d because rendered visual content can be represented as a multi-level set of raster representations; and constitutes a xe2x80x9cbrowsing systemxe2x80x9d because the client and server share data about the source visual content element being browsed, and the client performs a specific browsing function (assisted by the server). Remote browsing contrasts with remote frame-buffer or client/server display-list architectures, since in these architectures the client and server do not share data about the source visual content element and the client does not provide a specific browsing function. Remote browsing also contrasts with client-side browsing, where the client provides an integrated rendering and browsing function, while the server only provides access to the source visual content element.
Although the image is ultimately displayed on the client device according to the present invention, it is rasterized on the server side into a form compatible with the client display. The invention results in reduced software and hardware requirements on the client, centralized administration on the server, centralized rendering resources such as font libraries, improved security and reliability on the client, reduced power consumption for portable client devices, and optimizations for wireless data communications. The invention facilitates improved support for new types of client devices including PDAs such as the PALM PILOT, wireless telephones with bitmap displays, devices intended for use in automobiles, and devices embedded in household appliances. The original layout of the visual content is preserved across different devices with consistent rendering of various types of visual content.
The approach of the invention can be xe2x80x9cmulti-modalxe2x80x9d in the sense of providing or supporting multiple rendering modes, based on user preference and/or the type of content. For example, an implementation of the system may have two modes (a multi-level raster, and a single level text representation).
More generally, a rendering mode refers to the type of rendering technique being used, and the parameters supplied to a given technique. In the family of rendering solutions, a rendering technique can be defined by the following:
a) an optional pre-rendering step: this can provide a source-to-source transform (source transcoding and/or source extraction);
b) the rendering step: this provides the source-to-raster conversion; and
c) an optional post-rendering step: this provides optional raster-to-raster transforms (multi-level raster transforms and/or raster extraction).
The algorithms used in each step define different rendering modes. In addition, each of these steps can be performed on the client or the server. The different client/server combinations define different rendering modes, and a rendering step can also be split between server and client. For example, the server can render certain types of content while the client renders the rest. A visual content element may be described as a container of its constituent components, with each constituent component having a type (which might also be xe2x80x9ccontainerxe2x80x9d). Consequently, rendering by type can be applied on a xe2x80x9cper constituent componentxe2x80x9d basis.
Each step can have parameters. Different parameters generate different representations, but these need not constitute different rendering modes. For example, changing the font size parameter in a rasterized representation from 10 point Times to 12 point Helvetica creates a different look but is not a different rendering mode.
As noted above, xe2x80x9crenderingxe2x80x9d as used herein refers to generating displayable data from source dataxe2x80x94i.e., performing a source-to-raster conversion. A xe2x80x9csource,xe2x80x9d in this context, requires some type of interpretation or conversion to generate a raster representation. The term xe2x80x9ctranscodingxe2x80x9d is used herein to refer to source-to-source transforms. xe2x80x9cPixel transformsxe2x80x9d refer to raster-to-raster transforms, i.e., operations in which output pixel values can be computed from a corresponding region of input pixels. Standard image processing operations like convolution filters are xe2x80x9cpixel transforms,xe2x80x9d as are operations specific to the invention (e.g., generating a multi-level set of representations from a proxy display surface). The term xe2x80x9crasterizingxe2x80x9d refers to a process that uses whatever transcoding, rendering and/or pixel transforms are required to convert an input visual content element into a bitmap representation for display on a particular display device or class of display devices.
A xe2x80x9cpainting functionxe2x80x9d is the process of painting a bit-map representation onto a display screen using xe2x80x9cper-pixelxe2x80x9d operations (including simple scaling on a xe2x80x9cper pixelxe2x80x9d basis), and is therefore distinct from a pixel transform. A xe2x80x9cpainting functionxe2x80x9d is typically provided by a windowing system or other type of display manager. For example, in the context of the present invention, the server typically provides xe2x80x9cpixel transformsxe2x80x9d while the client provides the xe2x80x9cpainting functionxe2x80x9d to its bit-map display.
A representative implementation of multi-modal browsing combines multi-level server-side rendering with client-side rendering of a transcoded text-related source representation. These two different rendering modes are applied to the same visual content element. This implementation may also provide a means to share a single raster overview representation while browsing over two different modal representations of the detail (detail raster and a rendering of the text-related representation). Since the multi-level raster representations are generated on the server, this approach constitutes multi-modal multi-level remote browsing.
The multi-level raster may, for example, have two levels (overview and detail) or more than two levels (i.e., at least one intermediate level). The user can optionally select a specific xe2x80x9cregion of interestxe2x80x9d on an overview raster representation for viewing at the detail level. This allows the user to control how much of the detail is being sent from server to client. As part of the detail selection, the user can specify a specific rendering mode or let the system decide based on the type of content selected.
The choice of detail mode (raster, text or automatic) may be set through the user interface on the client device. If the user desires the multi-level raster version of the detail, the server sends the corresponding partial raster representation to client for browsing. The server need not send the entire raster detail, which provides a significant performance improvement over traditional content-delivery systems.
If the user desires a text-related version of the detail, the server sends the corresponding partial transcoded source to the client. For text mode, the client may render the text as a word-wrapped raster representation that fits the client viewport""s horizontal pixel width. The user scrolls up and down the selected text, rather than dragging it from side to side. This also provides improved performance (compared with sending the entire detail raster, or sending the entire source representation), and is also more convenient for reading just the text. The client-side rendering of the transcoded text-related representation may be capable of displaying text, text-related formatting (bold, italic, larger type for headers) and hyperlinks; indeed, the invention is amenable to rendering the entire proposed XHTML Basic standard.
As a result, the invention is capable of handling virtually any desktop page (in both raster and text mode, with a multi-level interface shared between raster and text mode) and simultaneously handle any page designed for a tiny screen. The invention can essentially extract any part of a desktop page and convert it into a representation suitable for cell phone displays.
It should be noted that the rendering of the text-related transcoded representation need not always be on the client. For example, it might be more convenient to render Kanji (or other languages like Arabic) on the server, while still providing a text-related representation (to extract the text, and render it to fit exactly the width of the client viewport, with scrolling as needed). A variation of xe2x80x9ctext modexe2x80x9d may include rasterized representations of graphics components (such as tables or forms). In this mode, graphics components are rendered (on the server or client) to fit within the horizontal width of the client viewport. The rendered graphics components are sequenced vertically with text-related components, creating a segmented vertical strip of rendered content.
Accordingly, in one aspect, the invention provides a method for displaying visual content on a client device. This method includes the steps of establishing a communications link between the client device and a server, determining a display attribute of the client device, representing a visual content element on a proxy display surface of the server, transforming the represented visual content element and transmitting the bitmap to the client device for display along with a browsing-related attribute of the visual content element. The browsing-related attribute can, for example, describe data indicating the source (e.g., location or type) of the visual content, or equivalent data for a constituent component of the visual content. In one embodiment, the representation of the visual content element includes rasterizing the visual content element on the proxy display surface of the server. In another embodiment, the method includes the additional step of generating a display surface in a display memory at the client device in response to the bitmap and the data element. In yet another embodiment, the method also includes the step of generating a user interface at the client device in response to the data element.
In another aspect, the invention provides a method for generating on a server a multi-level set of bitmaps from an input bitmap. The method includes the steps of determining a display attribute of a client device, selecting a pixel region of an input bitmap, performing a transform operation on the pixel region to generate a first bitmap, defining a data element corresponding to a second bitmap, and transmitting the first bitmap, the second bitmap and the data element to the client. The data element corresponding to the second bitmap defines a relationship of the first bitmap to the second bitmap. In one embodiment, the method includes the additional step of receiving an input bitmap at the client device. In another embodiment, the step of selecting includes recalling a stored selection identifying the pixel region.
In another aspect, the invention provides a method for generating on a server a multi-level set of bitmaps from an input bitmap. The method includes the steps of determining a display attribute of a client device, identifying multiple pixels regions of an input bitmap and performing a transform operation on each of the identified pixel regions to generate multiple bitmaps based on the display attribute of the client device. The method also includes the steps of defining a plurality of data elements and transmitting the bitmaps and the data elements to the client. Each data element corresponds to a respective bitmap and defines a relationship of the respective bitmap to the input bitmap. In one embodiment, the transform operation is a scaling operation.
Another aspect of the invention provides a server for providing a rasterized visual content element to a client device. The server includes a proxy module, a transformation module in communication with the proxy module, and a communications module in communication with the transformation module. The proxy module is used to represent a proxy display. The transformation module generates a bitmap in response to an input bitmap and a display attribute of the client device. The communications module transmits the bitmap to the client device for display. In one embodiment, the transformation module is a scaling module.
Still another aspect of the invention provides a server for generating a multi-level set of bitmaps from an input bitmap. The server includes a proxy module for representing a proxy display, a communications module to receive pixel region selection data from a client device, and a transformation module in communication with the proxy module and the communications module. The transformation module performs a transform operation on a pixel region of the input bitmap to generate a first bitmap in response to the pixel region selection data and generates a data element corresponding to a second bitmap. The data element defines a relationship of the first bitmap to the second bitmap. In one embodiment, the transformation module is a scaling module.
Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating the principles of the invention by way of example.